Skip to content

Capture funnel finished + meaning-preserving archival + AI composer (Epics 13/16/15) - #2

Merged
Seanathon merged 10 commits into
mainfrom
dev-capture-archive-compose
Jun 24, 2026
Merged

Capture funnel finished + meaning-preserving archival + AI composer (Epics 13/16/15)#2
Seanathon merged 10 commits into
mainfrom
dev-capture-archive-compose

Conversation

@Seanathon

Copy link
Copy Markdown
Owner

Capture funnel finished + Meaning-preserving archival + AI composer (views, not copies)

This branch lands 8 stories across three v2 epics, building on the Epics 12–14 spine already on main. Everything is additive — the no-regression constraint (NFR-BC) is enforced by a dedicated test per story, the single-FK item.board_id is preserved, and existing boards/items/assets are untouched.

Tests: npm test492 pass / 0 fail (was 423 on main), 0 regressions.

Stories included

# Story What it adds
13.3 PWA + Web Share Target Installable PWA (manifest.webmanifest + service worker) and a POST /share handler so any app's share sheet one-taps a URL into the Inbox
13.4 Browser-extension review lane A pure, token-authed API client + an MV3 popup that triages the Inbox via the AI suggested-board chip (one-tap confirm → the assign verb)
16.1 Snapshot asset (SingleFile) A kind='snapshot' self-contained-HTML asset captured on the existing single-Chrome sidecar, so a curated link survives the page going down
16.2 Opt-in archival trigger A default-off archive_on_promote board flag + a per-item archive action that fire the snapshot on the one assign verb
16.3 Archive footprint + backfill GET /api/archive/footprint (snapshot disk usage) + a serial, idempotent archive:backfill CLI
15.1 View-definition model A view table — a read-only cross-board lens (filter + optional pin/order overlay), never a copy
15.2 AI collection composer A propose-only skill that proposes home-board assignments and/or a view; accept reuses the one assign verb + the view model
15.3 Materialize view → board The copy-on-write escape hatch: copy a lens's items into a real board (assets reused by hash, source untouched)

One new dependency: single-file-cli@2.0.83 — added as an optionalDependency (it's AGPL-3.0; lazily import()-ed only when an archive runs, so board-oss core never loads AGPL code on a normal path). Passed the Socket score gate (supply_chain 0.81 / quality 0.99 / vuln 1.00 / maintenance 0.93).


How the new features work

Capture funnel (Epic 13, now complete)

  • PWA / share target (13.3): the manifest declares a share_targetPOST /share; the handler extracts the URL (from url, or the first URL in shared text/title) and saves it to the Inbox via the same cheap-capture create path as the API — no second capture path. The service worker is additive and passes /api, /events (SSE), /screenshots, /share straight through, so it can't buffer the live-status stream.
  • Browser extension (13.4): an MV3 popup (extension/) over a pure api-client.js (no DOM) that speaks only the /api/v1/* contracts. Each recent Inbox item shows its AI suggested-board chip; tapping it promotes via POST /api/v1/items/assign. Degrades to a manual board picker when there's no suggestion. The token lives in chrome.storage.local, sent only as a Bearer header (a remote instance must be https).

Meaning-preserving archival (Epic 16)

  • Snapshot (16.1): runs through the existing concurrency-1 Chrome sidecar (never a second Chromium), with a per-snapshot byte cap + timeout, hash-dedupe, and graceful degradation — a failed archival never flips a curated item to error (status-neutral).
  • Opt-in (16.2): off by default. Set a board's archive_on_promote flag (additive descriptor field) or call the per-item archive action — the snapshot fires on the one assign verb, alongside the AI takeaway (which lives on the item, not a copy).
  • Footprint + backfill (16.3): see how much disk archives use; backfill existing curated items serially through the one Chrome (idempotent by item id — safe to re-run/resume).

AI composer — views, not copies (Epic 15)

  • View (15.1): a "composed board" is a saved lens — a row of {filter, order?, captions?} that resolves dynamically (new matches auto-appear) across boards, read-only. Items keep one canonical home board.
  • Composer (15.2): describe a board and the AI proposes home-board assignments and/or a view — a reviewable proposal that persists nothing until you accept. Accept reuses the one assign verb + the view model; bounded validate-and-repair; degrades to a manual builder with no AI.
  • Materialize (15.3): when you want to hand-prune, copy a view into a real board — new rows, asset files reused by hash (no duplicate bytes), source untouched. Divergence is a deliberate choice.

Manual testing: UI vs API/CLI

✅ Usable in the UI / browser

  • Inbox + share target: load the app, then POST /share (or, on a real installed PWA, share a page) → the item lands in the Inbox.
  • Browser extension: load extension/ as an unpacked MV3 extension, set your instance URL + BOARD_API_TOKEN in its options, open the popup on any tab → save + triage. (Real install + OS share-sheet are manual/mobile QA — see "Staged" below.)

🔌 Usable via the API / CLI

TOKEN=your-token; BASE=http://localhost:3141
# share-to-Inbox (no auth; same-origin handler)
curl -s -X POST $BASE/share -H 'content-type: application/x-www-form-urlencoded' --data 'url=https://example.com'
# per-item archive (opt-in archival, 16.2)
curl -s -X POST $BASE/api/v1/items/<id>/archive -H "Authorization: Bearer $TOKEN"   # 202 / 404 / 422
# archive footprint (16.3)
curl -s $BASE/api/archive/footprint                                                  # {totalBytes,count}
# serial backfill (16.3) — stop the server first (concurrency-1 is per-process)
npm run archive:backfill
# composer (15.2) — propose-only; needs an LLM provider, else returns a draft
curl -s -X POST $BASE/skills/compose-collection -H 'content-type: application/json' -d '{"description":"a board about retrieval"}'

🚧 Staged / manual QA (documented per story, not claimed green)

  • PWA install + SW-vs-SSE runtime + OS share-sheet (13.3) — browser/OS-level, verified by inspection + live curl; the node suite proves additive serving + route preservation.
  • SingleFile CDP wiring (16.1) — the default driver (dynamic import + backEnd:'cdp' against the existing Chrome) is inspection/manual-verified; the suite fakes the capture and proves the asset/dedupe/degradation/no-regression logic.
  • Composer & view SPA mounting (15.2/15.3) — the read-model, resolver, propose/accept primitives are tested at the function level; mounting them in the SPA is staged DOM (the Epic 8 precedent).

Notes for review

  • Each story file in docs/bmad/stories/ is at Status: review with a full Dev Agent Record (decisions + the party-mode review findings fixed before commit).
  • Reviews caught + fixed, among others: a snapshot-job nested-enqueue deadlock on the success path (16.1); a typed-error-union violation + non-atomic accept in the composer (15.2); the shared-file delete guard's key mismatch (15.3); a cleartext-token vector in the extension (13.4); plus several confounded/vacuous tests replaced with discriminating ones.
  • Known follow-ups (documented, out of scope): an app-wide capture-layer SSRF denylist (pre-existing; affects every create path); deleteItemWithAssets resolves snapshots/* assets under the screenshots dir (Epic 16 didn't update the deleter); a materialized board's descriptor fields aren't editable (the fields:[] v1 choice).

Seanathon and others added 10 commits June 23, 2026 12:41
Make board-oss an installable PWA and register it as a native share target so
any app's share sheet can save a URL to the Inbox in one tap.

- manifest.webmanifest + SVG icons (any + maskable); <link rel=manifest>, theme-
  color, and a guarded service-worker registration in index.html's <head>.
- sw.js: minimal additive worker — shell-caches, but returns before respondWith
  for /api, /events (SSE), /screenshots, /share, and all non-GET, so by
  construction it never buffers the Story 5.3 live-fill stream.
- POST /share handler in its own encapsulated plugin (scoped urlencoded parser,
  no new dep; root JSON parser untouched). Reuses addItemSkill.run → no board →
  Inbox + cheap tier — the same create path /api/v1/items uses (the server holds
  only the token hash, so it cannot re-POST to the bearer-guarded API). Extracts
  the URL from url/text/title, strips trailing punctuation, returns the user.

Additive (NFR-BC): existing SPA/collection/item routes + SSE serve unchanged;
the urlencoded parser is proven scoped (root JSON route still 415). SW-vs-SSE
runtime + real install/share-sheet are documented manual Chrome/mobile QA.

Party-mode review (Winston/Amelia/Quinn) applied: trailing-punctuation strip,
enctype assertion, dead-drain cleanup. Pre-existing app-wide capture SSRF noted
as Epic-level backlog (not introduced here; /share is stricter than the existing
unauthed collections POST).

8 new tests; suite 431 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close Epic 13's capture funnel with a desktop review lane: a pure, token-authed
API client + an MV3 popup that surfaces each recent Inbox capture with its AI
suggested-board chip (one-tap confirm), degrading to a manual picker.

- extension/api-client.js: pure ESM (no DOM/chrome.*) — save/listRecent/
  getSuggestion/listBoards/assign + reviewAction, speaking ONLY the Epic 12+14
  /api/v1/* contracts. No server-side files; no second backend.
- popup.html/js: the review lane (chip→assign on confirm, manual picker fallback
  that also covers a failed suggestion call so an item is always promotable).
- options + manifest: instance URL + token in chrome.storage.local (Bearer-only,
  never logged/URL'd; remote instance must be https — no cleartext token leak).

Tests (7) are two-layered: fake-fetch contracts pin URL/Bearer/body, and an
inject-backed round-trip routes the client's fetch into a real buildServer to
prove save→Inbox and assign→board_id-moves against the LIVE contract (it caught
the real assign response shape). newest-first is honest passthrough (server-
ordered). No-regression: diff is extension/* + the test-script line only; assign
moves only on explicit confirm.

Party-mode review applied: https-only remote, always-promotable fallback,
getSuggestion/listBoards coverage. Suite 438 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a kind='snapshot' self-contained-HTML asset so a curated link's content
survives the page going down. Captured through the EXISTING single-Chrome
sidecar (no second Chromium), persisted ADDITIVELY (never wipes the screenshot).

- config: derived snapshotsDir under DATA_DIR + ensureDataDir mkdir.
- db/snapshot-asset.ts: writeSnapshotAssetDirect — additive single-row upsert on
  a stable ${id}-snapshot id (NOT writeItemDirect's replace-all), dedupe-read +
  upsert in one transaction; enqueued wrapper for standalone callers.
- capture/url-snapshot.ts: createUrlSnapshotCapture (byte cap + teardown) +
  runSnapshotJob — status-neutral (enqueueJob, never runItemJob, so a failed
  archive never flips a curated item to error); timeout SIGKILLs Chrome before
  the slot releases. Default SingleFile driver is dynamic-imported (AGPL isolated)
  and drives the existing browser via CDP — no 2nd Chrome.

single-file-cli@2.0.83 scored (supply_chain .81 / quality .99 / vuln 1.0 /
maint .93; AGPL) and added as an optionalDependency + lazy import.

Party-mode review caught a nested-enqueue deadlock on the success path (job
holding the writer slot re-entered enqueueTransaction) — fixed with the in-slot
direct write + a success-path regression test. No-regression test proves the
screenshot row AND file survive (two asset rows); dedupe asserts the skipped
write, not just one row. Suite 448 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make archival opt-in and tied to promotion, so a small box archives what was
curated — not every bucket link.

- descriptor: additive OPTIONAL archive_on_promote flag (default-off) +
  archivesOnPromote() reader; pre-wave descriptors validate + read off (no
  migration — the descriptor is a JSON blob).
- assign verb (the ONE path): after the moves + earned enrichment, if the TARGET
  board archives-on-promote, enqueue a 16.1 snapshot for each MOVED item
  (forward-only; never skipped/failed; fire-and-forget; graceful). The composer
  (15.2) inherits this with no second code path.
- per-item REST action POST /api/v1/items/:id/archive (not a skill): 404 unknown,
  422 no-source, 202 queued.
- enqueueSnapshot injectable threaded buildServer → v1 → assignItems so tests
  spy it and never launch Chrome.

The earned takeaway (item.fields) coexists with the snapshot asset — disjoint
state, proven across the real spy-LLM enrich+trigger seam. Default-off, batch,
and 422 paths covered. Suite 459 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface snapshot disk usage and let operators backfill existing curated items —
so "no storage limit" never becomes a silent surprise. Closes Epic 16.

- db/archive-footprint.ts: read-only archiveFootprint — total bytes over
  kind='snapshot' files only (stat-on-disk, missing→0, never throws); surfaced
  at GET /api/archive/footprint. Mutates nothing.
- db/archive-backfill.ts: backfillSnapshots — enqueues a snapshot for each
  eligible (archive-on-promote board) item lacking one, serially on the single
  concurrency-1 worker. Idempotent by item id (skips already-snapshotted), so
  re-runs/resume create no duplicates.
- db/archive-backfill-cli.ts: `npm run archive:backfill` operator runner (awaits
  the serial drain before closing; warns to stop the server first — the
  concurrency-1 guarantee is per-process).

Additive/read-only (NFR-BC); no-parallel-Chromium inherited from enqueueJob
(in-process). Party-mode review applied (non-vacuous no-regression + byte-level
exclusion tests; cross-process server-stop warning). Suite 465 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the additive `view` table — a composed board is a read-only LENS over canonical
items (filter + optional pin/order overlay stored in the view row), never a
duplicate pile. Confirmed with the maintainer first (workshop hinge #1: lens, not
join / not m2m).

- db/schema.ts + db/index.ts: additive `view` table {id, name, filter JSON,
  optional order, optional captions} in drizzle + raw BOOTSTRAP_SQL lockstep.
  "view"/"order" are SQL keywords → quoted in the raw DDL. item/board schemas
  byte-for-byte unchanged (no item→view FK, no join table). D12 held.
- db/view.ts: resolveView — two-path (FTS MATCH across boards when a text query is
  present; plain SELECT + structured predicates otherwise, so a no-query lens never
  dead-matches). Order overlay pins matching ids first. SELECT-only — edits at an
  item's home flow into every view (canonical meaning). All predicate values bound.
- db/search.ts: export toFtsPhrase (shared, no duplicate).

NFR-BC: additive table, fresh-DB boot verified, zero item-row migration. Tests are
discriminating (dynamic N→N+1; overlay pins a non-first id; read-only asserts row
byte-identity; two-descriptor cross-board render). Party-mode review applied.
Epic 15 Story 1 of 3. Suite 475 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The composer: describe (or let the AI infer) a board and it PROPOSES how to build
it from saved items — home-board assignments and/or a cross-board view — as a
reviewable proposal that persists NOTHING until accept.

- skills/compose-collection.ts: composeCollectionSkill (propose-only) +
  acceptComposerProposal (thin dispatcher). Accept reuses the ONE assign verb
  (14.2 assignItems) for assignments and the 15.1 createView for a view — no
  second move/enrichment path (D8). Both the description AND scraped item content
  are fenced as untrusted in the prompt.
- descriptor/guardrails.ts: extract a generic boundedRepair<T,V,E> (≤1 repair);
  validateAndRepair becomes a thin behavior-preserving wrapper. The composer
  reuses the same discipline with its own board-aware validator + ComposerError.
- db/view.ts: createView write primitive (additive, serialized).

Degrades to an editable draft with no AI (never a 500). Accept is atomic on board
validity (fail-fast before any move) and reversible. NFR-BC: a bystander item
keeps its home board. Party-mode review applied (typed-error union fix, atomic
accept, item-dedup, fencing/bystander tests). Source typechecks clean under
strict. Epic 15 Story 2 of 3. Suite 485 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deliberate escape hatch (D11): turn a read-only lens (15.1) into a real,
hand-prunable board by COPYING its currently-resolved items — never a move.

- db/materialize.ts: materializeView — resolveView → insertBoard → writeItem a
  NEW item row per item (new id, dest board, fields/notes/favorite by value).
  Asset rows reuse the source file by path+hash (ZERO file I/O — referenced, not
  rewritten; NFR-1). Source items/asset rows byte-for-byte unchanged.
- db/item-actions.ts: deleteItemWithAssets is now reference-aware — it won't
  unlink a file another asset row still resolves to (basename-keyed, matching the
  unlink key). Makes the shared materialized/source file safe to delete around.

Copies are independent (divergence owned by the copy). Destination descriptor is
minimal (fields:[]) — a documented v1 choice (notes/favorite editable, fields not;
no cross-source descriptor merge). Additive/NFR-BC (asserted +1 board/+N items/+N
assets, nothing else mutated). Party-mode review applied (guard-key alignment,
source-asset + additive assertions, edge tests). Source typechecks clean.
Epic 15 complete. Suite 492 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hots)

Epic 16 added snapshot assets under snapshots/<id>.html but deleteItemWithAssets
still resolved every asset as basename-under-screenshotsDir, so deleting an item
or board orphaned its archived .html. Route each asset to its own dir by the
stored path prefix (snapshots/ → the snapshots sibling of screenshotsDir), and
re-key the shared-file guard on the full relative path so guard and unlink agree
(a same-basename file in another dir is no longer a false share). No signature
change. Tests: snapshot .html is unlinked; cross-dir same-basename files don't
cross-delete. Suite 494 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single-Chrome / concurrency-1 guarantee is per-process, so running the
archive:backfill CLI alongside the live server put two Chromiums on the box
(the OOM NFR-1 exists to prevent) — previously only a doc warning. Add a
cross-process guard: isServerListening() probes the configured host:port and
the CLI refuses to start if a server answers, with a BOARD_ALLOW_CONCURRENT_CHROME=1
override for operators who accept the risk. Suite 497 pass / 0 fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Seanathon

Copy link
Copy Markdown
Owner Author

Pushed two follow-up fixes to the batch's own code (from the post-batch backlog ticket):

  • deleteItemWithAssets now cleans up snapshot assets — Epic 16 added snapshots/<id>.html but the deleter still resolved everything as basename-under-screenshots, orphaning archived .html on item/board delete. Now routes each asset to its dir by its stored path prefix; shared-file guard re-keyed on the full relative path. (b640655)
  • archive:backfill refuses to run while the server is up — the single-Chrome/concurrency-1 guarantee is per-process; the CLI now probes the configured port and bails (override BOARD_ALLOW_CONCURRENT_CHROME=1), instead of only a doc warning. (10029b5)

Suite 497 pass / 0 fail. The app-wide SSRF denylist (the remaining substantive backlog item) is going up as its own follow-up PR since it's pre-existing and touches code already on main.

Seanathon added a commit that referenced this pull request Jun 24, 2026
Consolidate the consciously-deferred items at batch close: repo loose ends (the
stashed 8.4 note, prune dev-epic-12-17), cross-cutting debt (app-wide capture
SSRF denylist; deleteItemWithAssets resolving snapshots/* under the wrong dir;
materialized fields not editable; backfill cross-process Chrome lock), manual QA
owed (PWA/SW-vs-SSE, SingleFile CDP wiring), and staged SPA wiring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seanathon added a commit that referenced this pull request Jun 24, 2026
…erialized-fields

§A housekeeping done; SSRF (PR #3), snapshot-delete + backfill-lock (PR #2), and
two source strict-type gaps (fc67fe2) resolved. Still open: materialized-board
field editability (design), zero-tsc-and-gate-CI, server.ts dead code (after PR #2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Seanathon
Seanathon merged commit bf4aa32 into main Jun 24, 2026
2 checks passed
@Seanathon
Seanathon deleted the dev-capture-archive-compose branch June 24, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant